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lit  ABSTRACT 

Our  goal  is  to  understand  the  neural  basis  of  perception  and  to  test 
computational  models  of  visual  processing.  This  year  we  have  been 
pursuing  3  lines  of  research:  a)  we  have  been  continuing  our  earlier 
studies  on  ’motion  capture',  shape  from  shading  and  stereopsis.  Our 
research  suggests  that  image  segmentation  (based  on  texture,  occlusicti 
can)  profoundly  influences  the  early  visual  processing  of  stereopsis; 
shape  from  shading  and  motion  correspondence;  b)  we  have  begun  to 
investigate  the  'filling  in'  of  'artificial  scotomas'  and  of  scotomas 
produced  by  brain  injury.  (Our  goal  has  been  to  understand  surface 
interpolation),  and;  c)  we  have  developed  a  new  psychophysical 
technique  for  isolating  and  studying  a  "fast"  contour  system  in  human 
vision;  a  system  that  might  correspond  to  the  magnocellular  pathway 
of  physiologists. 


IS.  NUMBEI;  PAGES 


16.  PRICE  C. 


'.I'M  griTi.'mrTniM'.'TXtmjmi  I 


^-7 


N$N  7SAO.OI-2BO'SSOO 


St*nb«rd  Form  >’78  (Rtv.  2*89) 

'  fH  R1  1092  ’ 


92-08985 


Summary  of  Research  Projects  (V.  S.  Ramachandran/AFOSR) 

Our  lab  has  been  continuing  our  earlier  studies  on  shape  from  shading,  "motion  capture"  and 
structure  from  motion  (described  in  our  annual  report  last  year).  This  year  we  have  also  begun  two 
new  projects  which  I  will  describe  here. 

1 .  Perceptual  "filling  in"  of  scotomas.  Patients  with  damage  to  the  visual  conex  are  ordinarily 
unaware  of  their  scotomas.  It  is  usually  assumed  that  the  scotoma  gets  "filled  in"  by  the  texture 
and  color  from  the  surround.  What  are  the  spatial  and  temporal  constraints  on  this  "filling  in" 
process?  And  what  are  the  underlying  neural  mechanisms? 

The  importance  of  the  "filling  in"  process  derives  from  the  fact  it  is  a  manifestation  of  a 
normal  visual  process  called  surface  interpolation.  (It  is  unlikely  that  the  visual  system  has  evolved 
mechanisms  specifically  to  cope  with  scotomas!)  \S^en  you  look  at  any  visual  scene  the  visual 
system  probably  begins  by  using  outlines  to  create  a  set  of  "cartoon  sketches"  of  the  image  and 
then  proceeds  to  "fill  in"  Ae  surface  attributes.  It  is  this  process  that  one  is  tapping  into  when 
studying  the  filling  in  of  scotomas. 

We  have  been  studying  filling  in  in  four  situations: 

1 .  The  natural  blind  spot; 

2.  "artificial"  scotomas  induced  by  adaptation; 

3 .  scotomas  caused  by  laser  induced  retinal  lesions; 

4 .  scotomas  produced  by  damage  to  the  visual  cortex. 

By  comparing  "filling  in"  in  these  four  situations  we  hope  to  gain  a  novel  insight  into  the 
neural  mechanisms  underlying  surface  interpolation. 

Our  results  are  described  in  the  two  enclosed  papers.  (One  is  in  press  for  the  May  issue  of 
Scientific  American  and  the  second  is  being  submitted  to  Nature.) 

Personnel  involved  in  these  studies  include: 

1 .  Dr.  Diane  Rogers-Ramachandran,  post-doctoral  fellow; 

2.  Daniel  Plummer  and  Steven  Cobb,  graduate  students 

2.  "Phantom  contours":  Separate  neural  mechanisms  for  seeing  surfaces  and  seeing  contours. 
These  experiments  are  being  conducted  by  Dr.  Rogers-Ramachandrii.  there  are  computational 
(e.g.  see  Grossberg  et  al)  and  physiological  (e.g.  see  Livingstone  &  Hubei)  reasons  for  believing 
that  when  you  look  at  an  object  its  outline  and  its  surface  attributes  (e.g.  color/texture)  are  extracted 
by  separate  pathways.  We  now  present  new  psychophysical  evidence  for  the  existence  of  such  a 
distinction.  (See  enclosed  summary  from  a  recent  b(X>k  chapter  and  enclosed  videotape  of 
"Phantom  Contours.")  The  videotape  is  being  shown  later  this  week  by  Dr.  Diane  Rogers- 
Ramachandran  at  the  Simon  and  Fraser  annual  conference  on  vision  (Vancouver,  British 
Columbia). 


RE:  Spatial  and  Temporal  Interpolation  in  Visual  Object  Recognition 

This  proposal  by  Dr.  Phillip  J.  Kellman  raises  some  interesting  questions  and  certainly 
deserves  to  be  considered  for  funding  by  the  Airforce.  I  shall  discuss  two  aspects  of  the  proposal: 

a.  Both  illusory  contours  and  "amodal"  contours  are  striking  perceptual  effects  but 
until  recendy  interest  in  them  has  been  confined  mainly  to  psychologists.  Fortunately,  during  the 
last  10  years  or  so  there  has  been  a  tremendous  resurgence  of  interest  in  these  effects  among  both 
AI  researchers  and  neurophysiologists.  Most  of  the  work  in  the  field,  however,  has  been  of  a 
phenomenological  nature  (e.g.  see  the  stunningly  beautiful  demonstrations  in  Kanizsa's  book). 
Although  these  demonstrations  were  very  instmctive  we  also  need  systematic  psychophysics  and 
Dr.  Kellman's  proposed  experiments  may  help  fulfill  this  need.  Many  of  his  experiments  are 
designed  to  explore  the  spatial  and  temporal  constraints  on  these  two  processes. 

b.  Dr.  Kellman  also  believes  that  these  two  visual  effects  -  illusory  contours  and 
"amodal  contours"  are  based  on  a  "common  underlying  process,"  and  many  of  his  experiments  are 
designed  to  explore  this  possibility.  Personally,  I  don’t  find  this  aspect  of  his  proposal  especially 
interesting.  It  is  not  clear  to  me  what  exacdy  the  author  means  by  "common  underlying  process." 
Given  the  obvious  fact  that  both  these  phenomena  are  concerned  with  interpolation  of  contours  it  is 
hardly  surprising  that  there  are  many  similarities  between  them  but  the  mere  existence  of 
similarities  does  not  establish  that  there  is  a  common  mechanism.  (For  example,  there  are  many 
similarities  between  apparent  motion  and  stereopsis  but  one  wouldn't  want  to  argue  that  these  two 
phenomena  have  a  common  underlying  mechanism.)  Also,  there  are  many  displays  in  which  the 
perception  of  illusory  contours  actually  "vetoes"  or  excludes  amodal  contours  and  vice  versa  (e.g. 
see  recent  article  by  Nakayama  et  al  on  "neon  spreading"  and  stereopsis  in  Perception).  And 
finally,  I  seem  to  recall  that  illusory  contours  disappear  at  equiluminance  whereas  amodal 
completion  remains  essentially  unaffected. 

Let  me  add,  however,  that  these  objections  do  not  in  any  way  detract  from  the  value  of  the 
authors  proposed  experiments.  There  is  hardly  any  systematic  psychophysics  on  illusory  contours 
and  so  anv  data  would  be  valuable.  Indeed,  many  of  the  author’s  experiments  are  very  cleverly 
designed  to  explore  the  information  processing  strategies  underlying  the  perception  of  illusory 
contours  -  a  subject  about  which  very  little  is  known. 

Qualification  of  the  Research  Team 

The  PI,  Dr.  Kellman,  has  a  long  record  of  publication  in  this  field.  He  is  a  competent  and 
skilled  investigator. 

Overall  competence:  top  30%. 
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Overview  of  Research  Goals 

Our  research  during  the  last  2  years  has  been  concerned  mainly  with  the  perception 
of  motion,  stereopsis,  and  shape-from-shading,  and,  especially,  with  interactions  between 
these  processes.  We  have  been  using  mainly  psychophysical  techniques  but  our  results 
have  inspired  several  new  computational  models  and  lead  to  interesting  proposals  for 
physiological  experiments  at  the  single  unit  level.  Indeed  one  of  our  goals  has  been  to 
develop  conceptual  links  between  psychophysics,  physiology  and  AI  -  three  disciplines 
which  are  now  beginning  to  show  some  promise  of  convergence,  (Ramachandran,  1989). 
The  UCSD  campus  is  an  ideal  place  for  such  convergence  to  occur  since  we  can  draw  upon 
the  expertise  of  individuals  working  on  all  these  different  aspects  of  vision,  (eg. 
P.sychophysics-R.M.  Boynton,  D.I.A,  MacLeod,  and  D.  Rogers-Ramachandran; 


Physiology-T.  Albright,  Marty  Sereno,  G.  Baylis;  Computational  Models-T.Sejnowski,  D. 
Zipser,  Margaret  Sereno,  P.  Kuby;  Theoretical  Approaches-F.H.C.  Crick  and  Pat 
Churchland). 

Our  lab  works  in  close  association  with  Dr.  Albright’s  group  at  the  Salk  and  some 
of  the  Dr.  Albrights  graduate  students  (especially  G.  Stoner  and  K.  Dobkins)  have  already 
started  looking  for  physiological  correlates  of  vi.sual  illusions  that  have  been  di.scovered  in 
our  lab.  We  have  also  begun  some  collaborative  PET  studies  (on  awake  human  patients) 
with  Dr.  M.  Raichle  and  Dr.  Steve  Petersen  at  the  University  of  Washington  (St.  Louis). 
Two  major  conclusions  have  emerged  from  our  psychophysical  work. 

1 .  For  solving  any  given  computational  problem  (e.g.  motion  correspondence, 
structure  from  motion,  stereopsis,  the  aperture  problem,  etc.)  the  visual  system  seems  to 
use  multiple  parallel  heuristics  instead  of  a  single  sophisticated  algorhithm.  Our  goal  will  be 
to  identify  these  heuristics  and  find  out  what  particular  natural  constraints  are  being 
exploited  by  them.  The  advantage  of  using  multiple  mechanisms,  of  course,  is  that  they 
enable  the  organism  to  tolerate  noisy  (even  camouflaged)  images  of  the  kind  it  would 
normally  encounter  in  the  natural  world. 

2.  Our  second  major  conclusion  concerns  the  important  role  of  image  segmentation 
in  visual  perception.  AI  researchers  (especially  David  Marr)  have  tended  to  regard 
segmentation  as  a  separate  and  more  complex  problem  from  what  they  call  'early  vision'  - 
i.e.  stereopsis,  motion  correspondence  shape  from  shading,  etc.  But  our  research  suggests 
that  this  view  needs  to  be  revised.  Segmentation  based  on  various  cues  such  as  occlusion 
(e.g.  illusory  contours)  can  profoundly  influence  the  early  processing  of  motion,  stereopsis 
and  shading  (Ramachandran,  1989c).  A  similar  point  has  also  recently  been  made  by  Ken 
Nakayama  (Harvard).  Tliis  implies  that  they  must  be  a  great  deal  of  "cross  talk"  between 
difficult  visual  modules.  Tlie  numerous  reciprocal  cross-connection  described  by  David 
Van  Essen  provide  a  suitable  substrate  for  mediating  these  interactions. 

I  will  now  describe  some  of  the  projects  that  were  begun  in  1989  and  will  consider 
how  they  illustrate  the  ideas  I  have  outlined  above. 

Motion  Capture. 

If  moving  illusory  contours  are  superimposed  on  a  grid  of  stationary  dots  the  dots 
also  appear  to  move  in  the  same  direction.  Tliis  is  one  example  of  a  new  class  of  effects 
that  we  have  dubbed  'motion  capture'.  (Ramachandran  and  Inada,  1985;  Ramachandran, 
1990a,  1990b).  The  unambiguous  motion  signals  derived  from  certain  salient  features  in 
the  images  (e.g.  in  this  case  the  illusory  contours)  are  blindly  attributed  to  any  finer 
stimulus  features  in  the  vicinity.  Numerous  computational  models  have  been  proposed 
recently  to  account  for  these  phenomena  (e.g.  Bulthoft  et  al,  1989,  Nature.  337,  549-553; 
Yuille  and  Grzywacz,  1988,  Nature.  JJJ,  71-74;  D.  Gilden,  1990,  ARVO;  Grossberg, 
1990,  ARVO). 

Another  example  of  this  effect  can  be  seen  if  sparse  black  dots  are  optically 
superimposed  on  a  yellow  square  that  is  displayed  against  an  equiluminous  grey 
background,  when  the  spots  are  moved  the  yellow  square  also  appears  to  move  in  the 
.same  direction.  (Ramachandran,  1990a). 

Conflicts  between  'motion'  and  'position'  signals. 

If  the  black  spots  are  moved  continuously  -  as  in  a  conveyor  belt  -  the  yellow 
square  docs  rifll  move  continuously.  Instead,  it  appears  to  move  for  a  short  distance  and 


'jumps  back'.  This  illusion  arises  because  even  though  the  motion  system  is  fooled  into 
thinking  that  the  square  is  moving,  there  is  another  system  that  simultaneously  informs  the 
brain  that  tlie  location  (or  position)  of  the  square  has  not  changed. 


We  di.scovered  that  if  a  whole  horizontal  row  of  squtires  is  used  instead  of  a  single 
square  then  the  whole  row  does  indeed  appear  to  move  continuously.  By  adjusting  the 
spacing  between  the  squares  we  were  able  to  measure  the  sensitivity  of  the  'position' 
system.  (Ramachandran  and  Kleffner,  ARVO;  1990). 

Selective  attention  modulates  motion  capture. 

We  l^egin  with  a  random  matrix  of  7  spots  on  the  CRT  -  switch  them  off  and 
replace  them  in  Frame  2  with  the  same  7  spots  shifted  horizontally  by  about  1®.  (Figure 
12).  Tlie  2  frames  are  cycled  continuously  so  what  you  see  is  continuous  left-right 
oscillation  of  the  spots.  We  then  simply  take  an  opaque  piece  of  masking  tape  and  paste  it 
on  the  CRT  so  that  it  completely  occludes  one  of  the  spots  in  frame  2.  (The  spot  if  still 
visible  in  frame  1,  of  course).  Not  surprisingly  you  continue  to  see  the  spots  in  the 
surround  oscillate  as  before  but  the  remarkable  thing  is  that  the  single  unpaired  spot  also 
continues  to  oscillate  -  it  moves  behind  the  tape  even  though  its  partner  in  frame  2  is 
invisible.  If  you  have  two  bits  of  tape  -  one  on  each  side  of  the  unpaired  spot  -  the  spot 
always  "chooses"  to  move  behind  the  one  on  the  right. 

What  caiKses  this  illusion?  Well,  perhaps  the  motion  system  -  the  magno-MT 
system  -  'sees'  all  7  spots  as  one  big  blob  and  extracts  the  motion  signal  from  the  whole 
blob  without  resolving  the  individual  .spots.  Tlie  motion  signal  is  then  blindly  attributed  to 
all  the  finer  features  in  the  images  -  including  the  single  unpaired  spot  in  the  middle  -  so  that 
it  appears  to  move  as  well  ('motion  capture'). 

Diane  Rogers- Ramachandran  u.sed  this  display  to  explore  the  effects  of  visual 
attention.  Instead  of  having  a  single  set  of  7  spots  she  had  twQ  sets  of  spots  moving  in 
opposite  directions.  If  the  conditions  are  right  then  what  you  see  is  two  planes  of  spots  - 
moving  in  opposite  directions,  the  dots  being  grouped  together  by  "common  fate".  (You 
can  also  voluntarily  focus  your  attention  on  one  plane  alone  and  exclude  the  other.)  But 
what  if  you  now  add  a  single  unpaired  spot  in  the  middle  that  has  no  partner  in  frame  2? 

(Of  course,  you  also  provide  two  bits  of  masking  tape  -  one  on  either  side  of  it.) 

Docs  the  single  spot  then  appear  to  move  leftward  or  rightward?  Or  do  the  motion 
signals  from  the  2  oppositely  moving  planes  of  dots  just  cancel  each  other  -  so  that  you 
don't  see  any  motion  in  the  unpaired  spot?  The  answer  is,  it  depends  on  what  you  attend 
to.  If  you  attend  to  both  planes  of  dots  simultaneou.sly  -  which  is  difficult  but  can  be  done  - 
you  do  indeed  see  the  single  .spot  just  blinking  on  and  off  without  moving.  But  if  you 
attend  to  one  dot  cluster  alone  the  unpaired  spot  moves  along  with  that  cluster.  If  you 
attend  to  the  rightward  moving  dots  it  moves  rightward  and  if  you  attend  to  the  leftward 
moving  cluster  it  moves  leftward.  This  is  really  quite  remarkable  for  it  .suggests  that 
motion  capture  can  be  strongly  modulated  by  visual  attention  -  it  cannot  be  based  on  a 
simple  spread  of  motion  signals  from  the  moving  elements  to  the  unpaired  spot. 

Motion  Coherence  and  Transparency. 

Movshon  and  Adelson  have  shown  that  if  two  moving  gratings  are  superimposed 
they  will  appear  to  cohere  and  move  in  a  single  direction  ('pattern  motion')  instead  of 
,  moving  independently  -  (component  motion).  We  have  now  discovered  that  this  tendency 
to  see  pattern  motion  depends  crucially  on  the  luminance  of  the  intersections  (.Stoner, 


Albright,  and  Ramachandran,  1990;  RamachanJran,  1989c).  Specifically,  we  find  that 
when  the  two  gratings  luminances  are  multiplied  (to  mimic  two  physically  transparent 
gratings  or  N.D.  filters)  then  component  motion  is  .seen.  Our  hypothesis  is  that  the  motion 
system  has  tacit  'knowledge'  of  tlie  physics  of  transparency. 

One  of  our  future  goals  will  be  to  test  this  hypothesis  further.  We  believe  that  other 
researchers  in  this  field  (tliere  are  over  two  dozen  groups  working  on  motion  coherence) 
have  mi.ssed  this  ob.servation  because  they  simply  added  the  2  gratings  luminance  and  did 
not  vary  tlie  intersection  luminance  separately. 

We  have  also  started  exploring  tlie.se  phenomena  at  the  level  of  single  cells  in  MT. 
This  will  form  part  of  Gene  Stoner's  Ph.D.  thesis. 

Shaue-from-Shading. 

Another  line  of  research  being  pursued  in  our  laboratory  concerns  shape-from- 
shading,  (Ramachandran,  1988b).  It  is  quite  remarkable  that  since  the  time  of  Leonardo  da 
Vinci  (who  first  thought  about  this  problem)  there  have  been  only  two  or  three  systematic 
studies  on  shape-from-shading. 

Work  in  our  lab  suggests  that  shading  is  a  primary  vi.sual  dimension  (like 
stereopsis,  motion,  color,  etc.)  and  that  it  is  probably  extracted  early  in  visual  processing. 
(Kleffner  and  Ramachandran,  ARVO,  1990;  Ramachandran,  Scientific  American.  1988.) 

We  also  find  that  the  extraction  of  shape-from-shading  is  strongly  affected  by  image 
segmentation  -  e.g.  by  illusory  contours.  Chromatic  borders  and  texture  borders  are, 
however,  ineffective. 

Finally,  we  find  that  the  motion  system  can  use  shading  as  an  input.  In 
collaboration  with  Tom  Albright  (Salk  Institute)  we  are  now  trying  to  explore  the 
physiological  correlates  of  this  effcct  by  recording  from  direction  selective  cells  in  MT.  It  is 
known,  for  example,  that  these  cells  will  respond  not  only  to  moving  luminance  edges  but 
also  texture-edges  of  the  kind  originally  used  by  Ramachandran  et  al  (1973)  and  G. 

Sperling  in  Psychophysical  experiments.  Would  these  cells  also  respond  to  motion  that  is 
defined  exclusively  by  Shape-from-shading? 

Visual  Search  in  the  Presence  of  conflicting  Depth  Cues. 

An  unsolved  problem  in  Psychology  concerns  the  manner  in  which  information 
from  different  sources  is  combined  in  the  brain  to  create  a  unified  picture  of  the  world.  For 
example,  the  relative  distances  between  objects  can  be  conveyed  by  many  different  'depth 
cues'  such  as  perspective,  stereopsis,  shading  and  occlusion.  How  are  these  cues 
subsequently  integrated  to  compute  a  3-D  representation  of  the  world?  We  explored  this  by 
measuring  reaction  times  for  detecting  targets  in  which  different  cues  were  either  consistent 
or  in  conflict  with  each  other.  Results  suggest  that  different  depth  cues  are  initially 
combined  in  a  common  "cue-invariant"  3-D  representation  of  the  world  and  that  visual 
search  for  a  salient  target  is  carried  out  in  this  3-D  representation  ratlier  than  at  an  earlier 
stage  in  visual  processing.  (Plummer,  Pashler,  and  Ram.achandran,  ARVO,  1990; 
Ramachandran,  Pashler,  and  Plummer,  Psychonomic  Society,  1989). 


